In-vehicle mid-air gesture?based interaction method, electronic apparatus, and system

ABSTRACT

This disclosure provides an in-vehicle mid-air gesture-based interaction method, an electronic apparatus, and a system, and relates to the field of intelligent vehicle technologies. The method includes: obtaining a first mid-air gesture detected by a camera; and starting, when a preset response operation corresponding to the first mid-air gesture matches a first user who initiates the first mid-air gesture, the preset response operation corresponding to the first mid-air gesture in response to the first mid-air gesture. The method can be used in an in-vehicle mid-air gesture-based interaction scenario, reduce a mid-air gesture operation rate, and improve driving safety and interaction experience.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2020/100079, filed on Jul. 3, 2020, the disclosure of which ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

This application relates to the field of intelligent vehicletechnologies, and in particular, to an in-vehicle mid-air gesture-basedinteraction method, an electronic apparatus, and a system.

BACKGROUND

Currently, with rapid development of vehicle technologies, vehicles arebecoming increasingly popular, in addition to becoming an importantmeans of transportation in people's lives. In addition, with an increaseof vehicle-mounted terminal screens, selectable vehicle functions usedare not limited to driving functions. Human-machine interaction is animportant aspect of an intelligent cabin in future vehicles. A user willno longer be limited to touch control on a touchscreen. mid-air gestureswill gradually be used in human-machine interactions in advancedvehicles.

An important aspect to vehicle control is safety. There is a need toensure safe driving of a vehicle while improving human-machineinteractions.

SUMMARY

This disclosure provides a mid-air gesture-based interaction solution,including a mid-air gesture-based interaction method, an electronicapparatus, and a system, to be used in a vehicle cabin scenario toimprove experience of interaction between a user and an electronicapparatus while ensuring vehicle driving safety.

It should be understood that the mid-air gesture-based interactionmethod provided in embodiments of this disclosure may be performed by anelectronic apparatus. The electronic apparatus may be an entire systemof a computing device, or may be some devices, for example, a chip, inthe computing device. Specifically, the electronic device may be aterminal, for example, a vehicle or a vehicle-mounted device (forexample, a vehicle-mounted machine), may be a system chip (also referredto as a system-on-a-chip, or referred to as an SoC chip) that can bedisposed in a terminal, or may be another chip, for example, an imageprocessing chip. In physical implementation, the another chip, forexample, the image processing chip, may or may not be integrated in thesystem chip.

According to a first aspect, this disclosure provides an in-vehiclemid-air gesture-based interaction method. The method may include:obtaining a first mid-air gesture detected by a camera; and starting,when a first response operation corresponding to the first mid-airgesture matches a first user who initiates the first mid-air gesture,the first response operation in response to the first mid-air gesture.

With the foregoing solution, an electronic apparatus can be preventedfrom directly responding to any mid-air gesture initiated by any user,to ensure driving safety and information security.

In a possible implementation, before the starting, when a presetresponse operation corresponding to the first mid-air gesture matches afirst user who initiates the first mid-air gesture, the preset responseoperation corresponding to the first mid-air gesture in response to thefirst mid-air gesture, the method further includes: obtaining a responseattribute of the first response operation corresponding to the firstmid-air gesture; obtaining a user attribute of the first user whoinitiates the first mid-air gesture; and determining, based on theresponse attribute of the first response operation and the userattribute of the first user, whether the first response operationmatches the first user, where the response attribute includes one ormore of a response type, a response level, a response priority, or aresponse location, and the user attribute includes one or more of a useridentity, a user location, or a user permission.

In a possible implementation, the obtaining a user attribute of thefirst user specifically includes: determining the user location of thefirst user based on a picture that corresponds to the first mid-airgesture and that is detected by the camera; and determining the useridentity of the first user based on the user location of the first user.

It should be understood that the location of the first user isdetermined by using N pictures that correspond to the first mid-airgesture and that are collected by the camera, where N is a positiveinteger, and the user identity of the first user may be determined basedon the location. The user identity may be a driver or a passenger.Specifically, a five-seat vehicle is used as an example. The passengermay be a passenger in a co-driver seat, or may be a passenger in a rearseat. The passenger in a rear seat may also be classified into apassenger in a rear-left seat, a passenger in a rear-right seat, and apassenger in a rear-middle seat, where left and right directions aredetermined by using an advancing direction of the vehicle as a referencedirection. Alternatively, each seat in the vehicle is numbered, and acorrespondence between the number and the user location is preset, todetermine the user location of the first user based on the picture. Withthe foregoing solution, implementation is easy, and an amount ofcalculation is reduced.

In a possible implementation, the obtaining a user attribute of thefirst user specifically includes: determining the user identity of thefirst user based on face information of an in-vehicle user and a picturethat corresponds to the first mid-air gesture and that is detected bythe camera.

The face information of the in-vehicle user may be feature pointinformation of a user face image, or may be a user face image.

In a possible implementation, when the response attribute of the firstresponse operation is a driving operation and the user identity of thefirst user is a driver, the first response operation matches the firstuser. On the contrary, when the response attribute of the first responseoperation is a driving operation and the user identity of the first useris a passenger other than a driver, the first response operation doesnot match the first user.

It should be understood that matching between specific responseattributes and specific user attributes may be preset. For example, thefollowing may be preset: When the first response operation is anentertainment operation, response is performed to any user identity.However, when the first response operation is a driving operation,response is performed only to a user identity such as a driver or asafety administrator. A specific setting time may be a time beforedelivery, a time when an upgrade package is downloaded for upgrading asystem, a time when a user performs setting, or the like. The settingmay be stored locally on the electronic apparatus, or may be stored on acloud server, and is obtained by the electronic apparatus from the cloudserver by using a network each time. Specifically, for example, thesetting may be obtained from the cloud server by using a telematics boxT-box.

In a possible implementation, the obtaining a user attribute of thefirst user specifically includes: determining the user location of thefirst user based on a picture that corresponds to the first mid-airgesture and that is detected by the camera. The starting, when a firstresponse operation corresponding to the first mid-air gesture matches afirst user who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture specificallyincludes: starting, when the first response operation corresponding tothe first mid-air gesture matches the first user who initiates the firstmid-air gesture, the first response operation based on the user locationof the first user in response to the first mid-air gesture.

In a possible implementation, the starting, when the first responseoperation corresponding to the first mid-air gesture matches the firstuser who initiates the first mid-air gesture, the first responseoperation based on the user location of the first user in response tothe first mid-air gesture specifically includes: starting, when thefirst response operation corresponding to the first mid-air gesturematches the first user who initiates the first mid-air gesture, thefirst response operation in a region corresponding to the user locationof the first user in response to the first mid-air gesture.

For example, if the first response operation is opening a vehiclewindow, a window next to the first user may be opened, and no operationis performed on a vehicle window near another user. For another example,if the first response operation is adjusting a volume, a volume of aspeaker near the first user may be adjusted, so that only the volumearound the first user is adjusted without affecting another user.

With the foregoing solution, adaptive response may be performed based ona location of a user who initiates a mid-air gesture, to avoid affectingexperience of another user.

In a possible implementation, before the starting, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture, the method furtherincludes: displaying a first display interface on a first display in avehicle. The starting, when the first response operation correspondingto the first mid-air gesture matches the first user who initiates thefirst mid-air gesture, the first response operation based on the userlocation of the first user in response to the first mid-air gesturespecifically includes: when the first response operation correspondingto the first mid-air gesture matches the first user who initiates thefirst mid-air gesture and the first display is not a display in front ofthe first user, displaying the first display interface on a seconddisplay in the vehicle in response to the first mid-air gesture, andmoving an indication icon on the first display interface based on amoving trajectory of the first mid-air gesture in a preset direction.

With the foregoing solution, a user who initiates a mid-air gesture cansense, in an accurate and timely manner, that the camera has detectedthe mid-air gesture of the user and a processor is responding to anoperation of the user, thereby preventing the user from initiatingmid-air gestures for a plurality of times, and improving interactionexperience of the user.

In a possible implementation, the starting, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture specificallyincludes: when a user permission threshold required for the firstresponse operation corresponding to the first mid-air gesture is lessthan or equal to the user permission of the first user who initiates thefirst mid-air gesture, starting the first response operation in responseto the first mid-air gesture.

For example, when the first response operation is a driving operation, auser permission required for the response operation may be set to ahighest level, to exclude a passenger other than the driver and avoidsafety accidents. For another example, when the first response operationis device adjustment, for example, air conditioner temperatureadjustment, a user permission required for the response operation may beset to a specific level, to reduce repeated air conditioner temperatureadjustment caused by unintentional arm swinging by a young child, or tryto avoid a case that a child adjusts an air conditioner temperature tobe excessively low, affecting health.

In a possible implementation, the method further includes: detecting asecond mid-air gesture by using the camera, where the second mid-airgesture is initiated by a second user, and duration between aninitiation time of the second mid-air gesture and an initiation time ofthe first mid-air gesture is less than a first preset threshold. Thestarting, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture specifically includes: when the user permission of the firstuser is higher than a user permission of the second user, and when thefirst response operation corresponding to the first mid-air gesturematches the first user who initiates the first mid-air gesture,preferentially starting the first response operation in response to thefirst mid-air gesture.

With the foregoing solution, a problem that a plurality of usersinitiate mid-air gestures within a same time period and the electronicapparatus does not know which mid-air gesture is to be responded to orperforms random response may be resolved by preferentially responding toa mid-air gesture initiated by a user with a higher priority, therebyimproving interaction experience of the user.

In a possible implementation, a control permission of the driver ishigher than a control permission of another passenger.

In a possible implementation, a control permission of a passenger whoseage is greater than a first threshold is higher than a controlpermission of a passenger whose age is less than a second threshold,where the first threshold is greater than the second threshold, and thefirst threshold and the second threshold are positive integers.

In a possible implementation, a control permission of a first passengeris higher than a control permission of a second passenger, where an ageof the first passenger is greater than an age of the second passenger.

In a possible implementation, a control permission of an owner of amotor vehicle is higher than a control permission of a user other thanthe owner of the motor vehicle, for example, a driver and/or anotherpassenger. With the foregoing solution, property safety of the owner ofthe motor vehicle can be ensured, and the vehicle can be prevented frombeing robbed or stolen.

In a possible implementation, before the starting, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture, the method furtherincludes: when a distance between a location of the first mid-airgesture and the camera is greater than a second preset threshold,displaying a prompt message on a display in the vehicle, where theprompt message is used to prompt the first user to initiate a mid-airgesture to another camera.

In a possible implementation, before the starting, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture, the method furtherincludes: when a projection ratio of the first mid-air gesture relativeto a reference plane of a first camera is less than a third presetthreshold, displaying a prompt message on a display in the vehicle,where the prompt message is used to prompt the first user to initiate amid-air gesture to another camera.

In a possible implementation, before the starting, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture, the method furtherincludes: when detection precision for the first mid-air gesture is lessthan a fourth preset threshold, displaying a prompt message on a displayin the vehicle, where the prompt message is used to prompt the firstuser to initiate a mid-air gesture to another camera.

For example, a response operation for the first mid-air gesture iswaking up an in-vehicle mid-air gesture function. The camera wakes upthe in-vehicle mid-air gesture function after detecting the firstmid-air gesture. After determining that mid-air gesture detectionprecision, of the camera, corresponding to a location at which the firstmid-air gesture is initiated is less than a preset threshold, theprocessor may prompt, by using a prompt message, the user to initiate amid-air gesture to another camera, to increase a response speed anddetection precision for subsequent mid-air gesture-based interaction.The another camera may be a camera other than the camera, or may be acamera with a best detection angle for the user.

According to a second aspect, an embodiment of this disclosure providesan in-vehicle mid-air gesture-based interaction apparatus. The apparatusincludes a transceiver unit and a processing unit. The transceiver unitis configured to obtain a first mid-air gesture detected by a camera.The processing unit is configured to start, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture.

In a possible implementation, the transceiver unit is further configuredto obtain a response attribute of the first response operationcorresponding to the first mid-air gesture; the transceiver unit isfurther configured to obtain a user attribute of the first user whoinitiates the first mid-air gesture; and the processing unit is furtherconfigured to determine, based on the response attribute of the firstresponse operation and the user attribute of the first user, whether thefirst response operation matches the first user, where the responseattribute includes one or more of a response type, a response level, aresponse priority, or a response location, and the user attributeincludes one or more of a user identity, a user location, or a userpermission.

In a possible implementation, the processing unit is further configuredto determine the user location of the first user based on a picture thatcorresponds to the first mid-air gesture and that is detected by thecamera; and the processing unit is further configured to determine,based on the user location of the first user, whether the user identityof the first user is a driver.

In a possible implementation, when the response attribute of the firstresponse operation is a driving operation and the user identity of thefirst user is a driver, the first response operation matches the firstuser.

In a possible implementation, the processing unit is further configuredto determine the user location of the first user based on a picture thatcorresponds to the first mid-air gesture and that is detected by thecamera; and that the processing unit is configured to start, when afirst response operation corresponding to the first mid-air gesturematches a first user who initiates the first mid-air gesture, the firstresponse operation in response to the first mid-air gesture specificallymeans that: the processing unit is configured to start, when the firstresponse operation corresponding to the first mid-air gesture matchesthe first user who initiates the first mid-air gesture, the firstresponse operation based on the user location of the first user inresponse to the first mid-air gesture.

In a possible implementation, that the processing unit is configured tostart, when the first response operation corresponding to the firstmid-air gesture matches the first user who initiates the first mid-airgesture, the first response operation based on the user location of thefirst user in response to the first mid-air gesture specificallyincludes:

the processing unit is configured to start, when the first responseoperation corresponding to the first mid-air gesture matches the firstuser who initiates the first mid-air gesture, the first responseoperation in a region corresponding to the user location of the firstuser in response to the first mid-air gesture.

In a possible implementation, the processing unit is further configuredto display a first display interface on a first display in a vehicle;and

that the processing unit is configured to start, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture specifically meansthat:

the processing unit is configured to: when the first response operationcorresponding to the first mid-air gesture matches the first user whoinitiates the first mid-air gesture and the first display is not adisplay in front of the first user, display the first display interfaceon a second display in the vehicle in response to the first mid-airgesture, and move an indication icon on the first display interfacebased on a moving trajectory of the first mid-air gesture in a presetdirection.

In a possible implementation, that the processing unit is configured tostart, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture specifically means that:

the processing unit is configured to: when a user permission thresholdrequired for the first response operation corresponding to the firstmid-air gesture is less than or equal to the user permission of thefirst user who initiates the first mid-air gesture, start the firstresponse operation in response to the first mid-air gesture.

In a possible implementation, the transceiver unit is further configuredto obtain a second mid-air gesture detected by the camera, where thesecond mid-air gesture is initiated by a second user; and

that the processing unit is configured to start, when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture specifically meansthat:

the processing unit is configured to: when the user permission of thefirst user is higher than a user permission of the second user, and whenthe first response operation corresponding to the first mid-air gesturematches the first user who initiates the first mid-air gesture,preferentially start the first response operation in response to thefirst mid-air gesture.

In a possible implementation, a control permission of the driver ishigher than a control permission of another passenger.

In a possible implementation, the processing unit is further configuredto: when a distance between a location of the first mid-air gesture andthe camera is greater than a second preset threshold, display a promptmessage on a display in the vehicle, where the prompt message is used toprompt the first user to initiate a mid-air gesture to another camera.

In a possible implementation, the processing unit is further configuredto: when a projection ratio of the first mid-air gesture relative to areference plane of a first camera is less than a third preset threshold,display a prompt message on a display in the vehicle, where the promptmessage is used to prompt the first user to initiate a mid-air gestureto another camera.

In a possible implementation, the electronic apparatus is avehicle-mounted machine, a vehicle-mounted chip, or an intelligentvehicle.

According to a third aspect, an embodiment of this disclosure providesan intelligent vehicle, including the electronic apparatus according toany one of the second aspect or the possible implementations of thesecond aspect.

According to a fourth aspect, a terminal is provided. The terminalincludes: a touchscreen, where the touchscreen includes atouch-sensitive surface and a display; a camera; one or more processors;a memory; a plurality of application programs; and one or more computerprograms. The one or more computer programs are stored in the memory,the one or more computer programs include instructions, and when theinstructions are executed by the terminal, the terminal is enabled toperform the image processing method according to any one of the firstaspect and the second aspect or the possible implementations.

According to a fifth aspect, a computer storage medium is provided,including computer instructions, where when the computer instructionsare run on an electronic device, the electronic device is enabled toperform the interaction method according to any one of the first aspector the possible implementations.

According to a sixth aspect, a computer program product is provided,where when the computer program product is run on a computer, theelectronic device is enabled to perform the interaction method accordingto any one of the first aspect or the possible implementations.

According to a seventh aspect, a system is provided, including theelectronic apparatus according to any one of the second aspect or thepossible implementations of the second aspect, the intelligent vehicleaccording to any one of the third aspect or the possible implementationsof the third aspect, or the terminal according to any one of the fourthaspect or the possible implementations of the fourth aspect.

For technical details, technical effects, and possible implementationsof any one of the second aspect to the seventh aspect or the possibleimplementations of the second aspect to the seventh aspect, refer torelated descriptions in any one of the first aspect or the possibleimplementations of the first aspect. Details are not described hereinagain.

It should be understood that, with the in-vehicle mid-air gesture-basedinteraction method provided in embodiments of this disclosure, throughmatching between a user who initiates a mid-air gesture and a responseoperation corresponding to the mid-air gesture, interaction experienceof the operation can be ensured while safe driving is ensured.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a structure of an interior of a vehicleaccording to an embodiment of this disclosure;

FIG. 2A is a functional block diagram of a vehicle 200 according to anembodiment of this disclosure;

FIG. 2B is a schematic diagram of a photographing principle of a cameraaccording to an embodiment of this disclosure;

FIG. 3 is a schematic flowchart of an in-vehicle mid-air gesture-basedinteraction method according to an embodiment of this disclosure;

FIG. 4 is a schematic diagram of an in-vehicle mid-air gesture accordingto an embodiment of this disclosure;

FIG. 5 is a schematic diagram of modeling of an in-vehicle mid-airgesture according to an embodiment of this disclosure;

FIG. 6 is a schematic diagram of a scenario of an in-vehicle mid-airgesture according to an embodiment of this disclosure;

FIG. 7 is a schematic diagram of a structure of an electronic apparatusaccording to an embodiment of this disclosure;

FIG. 8 is a schematic diagram of a structure of another electronicapparatus according to an embodiment of this disclosure; and

FIG. 9 is a schematic diagram of a structure of another electronicapparatus according to an embodiment of this disclosure.

DESCRIPTION OF EMBODIMENTS

The following describes in detail embodiments of this disclosure withreference to the accompanying drawings.

FIG. 1 is a schematic diagram of a structure of an interior of a vehicleaccording to an embodiment of this disclosure. Currently, in the fieldof vehicles, a vehicle-mounted terminal, for example, a vehicle-mountedmachine (also referred to as an in-vehicle audio and video entertainmentsystem), may be fixed in a center console of a vehicle, and a screen ofthe vehicle-mounted terminal may also be referred to as a center consoledisplay or a center console screen. In addition, fully digital displayis gradually implemented in a cabin of some high-end vehicles, and oneor more displays are disposed in the cabin to display a digitaldashboard, an in-vehicle entertainment system, and other content. Asshown in FIG. 1 , a plurality of displays are disposed in a cabin, forexample, a digital dashboard display 101, a center console screen 102, adisplay 103 in front of a passenger in a co-driver seat (also referredto as a passenger in a front seat), a display 104 in front of apassenger in a rear-left seat, and a display 105 in front of a passengerin a rear-right seat. In addition, in FIG. 1 , although only one camera106 is shown near an A-pillar (pillar) on a driver side, a plurality ofcameras may be disposed in the cabin, and locations of the cameras areflexible. For example, a camera in the cabin may be disposed above thecenter console screen of the vehicle, a camera in the cabin may bedisposed on the left of the center console screen of the vehicle, acamera in the cabin may be disposed on the A-pillar or a B-pillar, and acamera in the cabin may be disposed in the front of a cabin roof of thevehicle. A user in the vehicle may initiate a mid-air gesture by usingthe camera to perform a corresponding function. However, a vehicle is aservice scenario highly sensitive to safety. How to ensure safe drivingof a vehicle while improving interaction experience is worth studying.

FIG. 2A is a functional block diagram of a vehicle 200 according to anembodiment of this disclosure. The vehicle 200 may include a pluralityof subsystems, for example, a travel system 202, a sensor system 204, acontrol system 206, one or more peripheral devices 208, a power supply210, a computer system 212, and a user interface 216. Optionally, thevehicle 200 may include more or fewer subsystems, and each subsystem mayinclude a plurality of elements. In addition, the subsystems and theelements of the vehicle 200 may be all interconnected in a wired orwireless (for example, Bluetooth) manner.

The travel system 202 may include a component that provides power formotion of the vehicle 200. In an embodiment, the propulsion system 202may include an engine 218, an energy source 219, a transmissionapparatus 220, and a wheel/tire 221. The engine 218 may be an internalcombustion engine, an electric motor, an air compression engine, or acombination of other types of engines, for example, a hybrid engineincluding a gasoline engine and an electric motor, or a hybrid engineincluding an internal combustion engine and an air compression engine.The engine 218 converts the energy source 219 into mechanical energy.

Examples of the energy source 219 include gasoline, diesel, otherpetroleum-based fuels, propane, other compressed gas-based fuels,anhydrous alcohol, a photovoltaic module, a battery, and other powersources. The energy source 219 may also provide energy for anothersystem of the vehicle 200.

The transmission apparatus 220 may transmit mechanical power from theengine 218 to the wheel 221. The transmission apparatus 220 may includea gearbox, a differential, and a drive shaft. In an embodiment, thetransmission apparatus 220 may further include another device, forexample, a clutch. The drive shaft may include one or more shafts thatmay be coupled to one or more wheels 221.

The sensor system 204 may include several sensors that sense informationabout a surrounding environment of the vehicle 200. For example, thesensor system 204 may include a positioning system 222 (the positioningsystem may be a GPS system, a BeiDou system, or another positioningsystem), an inertial measurement unit (inertial measurement unit, IMU)224, a radar 226, a laser rangefinder 228, and a camera 230. The sensorsystem 204 may further include a sensor that monitors an internal systemof the vehicle 200 (for example, a vehicle-mounted air quality monitor,a fuel gauge, or an oil temperature gauge). Sensor data from one or moreof these sensors may be used to detect an object and correspondingfeatures (a location, a shape, a direction, a speed, and the like) ofthe object. The detection and recognition are key functions forimplementing a safe operation by the autonomous vehicle 200.

The positioning system 222 may be configured to estimate a geographicallocation of the vehicle 200. The IMU 224 is configured to sense alocation and direction change of the vehicle 200 based on an inertialacceleration. In an embodiment, the IMU 224 may be a combination of anaccelerometer and a gyroscope.

The radar 226 may use a radio signal to sense an object in a surroundingenvironment of the vehicle 200. In some embodiments, in addition tosensing an object, the radar 226 may be further configured to sense aspeed and/or an advancing direction of the object.

The laser rangefinder 228 may use laser light to sense an object in anenvironment in which the vehicle 200 is located. In some embodiments,the laser rangefinder 228 may include one or more laser sources, a laserscanner, one or more detectors, and another system component.

The camera 230 may be configured to capture a plurality of images of asurrounding environment of the vehicle 200 or inside the vehicle. Thecamera 230 may be a still camera or a video camera. The camera 230 mayalso be referred to as a camera 230. An in-vehicle camera may be acamera in a driver monitor system (driver monitor system, DMS), or maybe a camera in a cabin monitor system (cabin monitor system, CMS), or abuilt-in camera of a vehicle-mounted computer.

The control system 206 controls operations of the vehicle 200 andcomponents of the vehicle 200. The control system 206 may includevarious elements, including a steering system 232, a throttle 234, abraking unit 236, a sensor fusion algorithm 238, a computer visionsystem 240, a path control system 242, and an obstacle avoidance system244.

The steering system 232 may operate to adjust an advancing direction ofthe vehicle 200. For example, in an embodiment, the steering system 232may be a steering wheel system.

The throttle 234 is configured to control an operating speed of theengine 218, so as to control a speed of the vehicle 200.

The braking unit 236 is configured to control the vehicle 200 todecelerate. The braking unit 236 may slow down the wheel 221 throughfriction. In another embodiment, the braking unit 236 may convertkinetic energy of the wheel 221 into a current. Alternatively, thebraking unit 236 may reduce a rotational speed of the wheel 221 inanother form, so as to control the speed of the vehicle 200.

The computer vision system 240 may operate to process and analyze animage captured by the camera 230, so as to recognize objects and/orfeatures in the surrounding environment of the vehicle 200. The objectsand/or the features may include a traffic signal, a road boundary, andan obstacle. The computer vision system 240 may use an objectrecognition algorithm, a structure from motion (Structure from Motion,SFM) algorithm, video tracking, and other computer vision technologies.In some embodiments, the computer vision system 240 may be configured todraw a map for an environment, track an object, estimate a speed of anobject, and the like.

The path control system 242 is configured to determine a travel path forthe vehicle 200. In some embodiments, the path control system 242 maydetermine the travel path for the vehicle 200 with reference to datafrom the sensor 238, the GPS 222, and one or more predetermined maps.

The obstacle avoidance system 244 is configured to recognize, evaluate,and avoid or bypass, in another manner, a potential obstacle in theenvironment of the vehicle 200.

Certainly, in an example, the control system 206 may additionally oralternatively include components other than those shown and described.Alternatively, some of the components shown above may be omitted.

The vehicle 200 interacts with an external sensor, another vehicle,another computer system, or a user by using the peripheral device 208.The peripheral device 208 may include a wireless communications system246, a vehicle-mounted computer 248, a microphone 250, and/or a speaker252.

In some embodiments, the peripheral device 208 provides a means for auser of the vehicle 200 to interact with the user interface 216. Forexample, the vehicle-mounted computer 248 may provide information forthe user of the vehicle 200. The user interface 216 may further operatethe vehicle-mounted computer 248 to receive a user input. Thevehicle-mounted computer 248 may be operated by using a touchscreen.Sometimes, the vehicle-mounted computer 248 may also be referred to as avehicle-mounted machine or an in-vehicle infotainment system. In othercases, the peripheral device 208 may provide a means for the vehicle 200to communicate with another device located in the vehicle. For example,the microphone 250 may receive audio (for example, a voice command oranother audio input) from the user of the vehicle 200. Likewise, thespeaker 252 may output audio to the user of the vehicle 200.

The wireless communications system 246 may wirelessly communicate withone or more devices directly or by using a communications network. Forexample, the wireless communications system 246 may use 3G cellularcommunication such as CDMA, EVD0, or GSM/GPRS, 4G cellular communicationsuch as LTE, or 5G cellular communication. The wireless communicationssystem 246 may communicate with a wireless local area network (wirelesslocal area network, WLAN) by using Wi-Fi. In some embodiments, thewireless communications system 246 may directly communicate with adevice by using an infrared link, Bluetooth, or ZigBee. Other wirelessprotocols, such as various vehicle communications systems, for example,the wireless communications system 246, may include one or morededicated short range communication (dedicated short rangecommunication, DSRC) devices, and these devices may include publicand/or private data communication between vehicles and/or roadsidestations.

The power supply 210 may supply power to the components of the vehicle200. In an embodiment, the power supply 210 may be a rechargeablelithium-ion or lead-acid battery. One or more battery packs of such abattery may be configured as a power supply to supply power to thecomponents of the vehicle 200. In some embodiments, the power supply 210and the energy source 219 may be implemented together as thoseimplemented together in some all-electric vehicles.

Some or all functions of the vehicle 200 are controlled by the computersystem 212. The computer system 212 may include at least one processor213. The processor 213 executes instructions 215 stored in anon-transitory computer-readable medium such as a data storage apparatus214. The computer system 212 may be alternatively a plurality ofcomputing devices that control individual components or subsystems ofthe vehicle 200 in a distributed manner.

The processor 213 may be any conventional processor such as acommercially available CPU. Alternatively, the processor may be adedicated device such as an ASIC or another hardware-based processor.Although FIG. 2A functionally shows the processor, the memory, and otherelements of the computer system 210 in a same block, a person ofordinary skill in the art should understand that the processor, thecomputer, or the memory may actually include a plurality of processors,computers, or memories that may or may not be stored in a same physicalhousing. For example, the memory may be a hard disk drive or anotherstorage medium located in a housing different from that of the computer210. Therefore, a reference to the processor or the computer isunderstood as including a reference to a set of processors, computers,or memories that may or may not operate in parallel. Different fromusing a single processor to perform the steps described herein, somecomponents, such as a steering component and a deceleration component,each may have its own processor, and the processor performs onlycalculation related to a component-specific function.

In various aspects described herein, the processor may be located awayfrom the vehicle and wirelessly communicate with the vehicle. In anotheraspect, some of the processes described herein are performed on aprocessor arranged in the vehicle, and others are performed by a remoteprocessor, including performing a necessary step to perform singlemanipulation.

In some embodiments, the data storage apparatus 214 may include theinstructions 215 (for example, program logic), and the instructions 215may be executed by the processor 213 to perform various functions of thevehicle 200, including the functions described above. The data storageapparatus 214 may also include additional instructions, includinginstructions to send data to, receive data from, interact with, and/orcontrol one or more of the propulsion system 202, the sensor system 204,the control system 206, and the peripheral device 208.

In addition to the instructions 215, the data storage apparatus 214 mayfurther store data, for example, a road map, path information, alocation, a direction, a speed, and other vehicle data of a vehicle, andother information. The information may be used by the vehicle 200 andthe computer system 212 when the vehicle 200 operates in an autonomousmode, a semi-autonomous mode, and/or a manual mode.

The user interface 216 is configured to provide information for orreceive information from the user of the vehicle 200. Optionally, theuser interface 216 may include one or more input/output devices in a setof peripheral devices 208, for example, the wireless communicationssystem 246, the vehicle-mounted computer 248, the microphone 250, andthe speaker 252.

The computer system 212 may control a function of the vehicle 200 basedon inputs received from various subsystems (for example, the travelsystem 202, the sensor system 204, and the control system 206) and theuser interface 216. For example, the computer system 212 may control, byusing an input from the control system 206, the steering unit 232 toavoid obstacles detected by the sensor system 204 and the obstacleavoidance system 244. In some embodiments, the computer system 212 isoperable to provide control over many aspects of the vehicle 200 and thesubsystems of the vehicle 200.

Optionally, one or more of the foregoing components may be mounted inseparation from or associated with the vehicle 200. For example, thedata storage apparatus 214 may be partially or completely separated fromthe vehicle 200. The foregoing components may be communicatively coupledtogether in a wired and/or wireless manner.

Optionally, the foregoing components are merely examples. In actualapplication, components in the foregoing modules may be added or omittedaccording to an actual requirement. FIG. 2A should not be understood asa limitation on embodiments of this disclosure.

An automated driving vehicle traveling on a road, for example, theforegoing vehicle 200, may recognize an object in a surroundingenvironment of the vehicle, to determine whether to adjust a currentspeed. The object may be another vehicle, a traffic control device, oranother type of object. In some examples, each recognized object may beindependently considered, and a speed to which the automated drivingvehicle needs to be adjusted may be determined based on a feature of theobject, for example, a current speed or an acceleration of the object,or a distance between the object and the vehicle.

Optionally, the automated driving vehicle 200 or computing devices (forexample, the computer system 212, the computer vision system 240, andthe data storage apparatus 214 in FIG. 2A) associated with the automateddriving vehicle 200 may predict behavior of a recognized object based ona feature of the recognized object and a condition (for example,traffic, rain, or ice on a road) of a surrounding environment.Optionally, all recognized objects depend on behavior of each other.Therefore, all the recognized objects may be alternatively jointlyconsidered to predict behavior of a single recognized object. Thevehicle 200 can adjust the speed of the vehicle 200 based on thepredicted behavior of the recognized object. In other words, theautomated driving vehicle can determine, based on the predicted behaviorof the object, that the vehicle needs to be adjusted to a stable status(for example, an adjustment operation may include acceleration,deceleration, or stop). In this process, another factor may also beconsidered to determine the speed of the vehicle 200, for example, alateral location of the vehicle 200 on a road on which the vehicle 200travels, a curvature of the road, and proximity between a static objectand a dynamic object.

In addition to providing an instruction for adjusting the speed of theautomated driving vehicle, the computing device may further provide aninstruction for modifying a steering angle of the vehicle 200, so thatthe automated driving vehicle follows a given trajectory and/ormaintains a safe lateral distance and a safe longitudinal distance froman object (for example, a car in an adjacent lane of the road) near theautomated driving vehicle.

The vehicle 200 may be a car, a truck, a motorcycle, a bus, a boat, anairplane, a helicopter, a lawn mower, an entertainment vehicle, aplayground vehicle, a construction device, a tram, a golf cart, a train,a handcart, or the like. This is not particularly limited in embodimentsof the present invention.

FIG. 2B is a schematic diagram of a photographing principle of a cameraaccording to an embodiment of this disclosure. As shown in FIG. 2B, thecamera 230 usually includes a lens (lens) and a photosensitive element(sensor). The photosensitive element may be any photosensitive device,for example, a CCD (charge-coupled device, charge-coupled device) or aCMOS (complementary metal oxide semiconductor, complementary metal oxidesemiconductor). Still as shown in FIG. 2B, in a process of taking aphoto or recording a video, an optical image may be generated afterreflected light of a photographed object passes through the lens, theoptical image is projected onto the photosensitive element, thephotosensitive element converts a received optical signal into anelectrical signal, and then the camera 230 sends the obtained electricalsignal to a DSP (Digital Signal Processing, digital signal processing)module for digital signal processing, to finally obtain each frame ofdigital image. In this embodiment of this disclosure, the camera 230 maybe configured to collect a mid-air gesture input by a user. For example,the camera 230 may be set to an always on (always on) state to collect acurrent image in real time. The camera 230 may send each frame ofcollected image to a processor 213. If it is recognized, from thecollected image, that the user has performed a preset enabling gesture,it indicates that the user expects to interact with an electronic deviceby using a mid-air gesture. In this case, the processor 110 mayrecognize, based on an image sent by the camera 230 in real time, aspecific mid-air gesture input by the user in this case.

FIG. 3 is a schematic flowchart of an in-vehicle mid-air gesture-basedinteraction method according to an embodiment of this disclosure. Itshould be understood that the interaction method may be performed by anelectronic apparatus. The electronic apparatus may be an entirecomputing device, for example, a vehicle or a vehicle-mounted devicesuch as a vehicle-mounted machine, or may be some components used in acomputing device, for example, a chip in a vehicle-mounted machine. Asshown in FIG. 3 , the method includes the following steps.

S301: Obtain a first mid-air gesture detected by a camera.

At least one camera may be mounted in a vehicle, for example, a cameraof a DMS, a camera of a CMS, a camera mounted on a vehicle-mountedmachine, or a camera mounted near a center console. The camera of theDMS may be mounted near an A-pillar (A-pillar) in the vehicle or near alocation of a dashboard. Usually, the camera of the DMS needs to be ableto collect an expression, a posture, an action, and the like of adriver, to analyze and determine behavior of the driver, so as to assistthe driver in behavior, for example, driving. Usually, the camera of theCMS may be mounted near an upper center region of a front windshield,and/or in a top center region of a cabin or at another location. Aphotographed target of the camera of the CMS usually includes a user inthe cabin, for example, a driver, a passenger in a front seat, or apassenger in a rear seat. The camera of the CMS usually needs to collectan expression, a posture, an action, and the like of the user in thecabin, to monitor behavior of the user in the cabin.

The camera may be a multi-lens camera, for example, a single-lenscamera, a dual-lens camera, or a triple-lens camera, or may be a 3Dcamera. A processor may obtain, based on an image obtained by thecamera, a change in a location of an object or a user in the image. Forexample, the processor may obtain, based on an image obtained by the 3Dcamera, a change in a distance between an object or a user in the imageand a vehicle-mounted machine. Alternatively, the processor may obtainimages collected by a plurality of cameras, and obtain, by building amodel, a change in a distance between an object or a user in the imagesand the vehicle-mounted machine.

Data collected by the camera may be stored in a memory, or uploaded to acloud for storage. The processor may obtain, by using an interfacecircuit through wired communication, wireless communication (forexample, Bluetooth or a cellular network), or the like, the datacollected by the camera.

For example, an in-vehicle camera may be set to a normally open state.In other words, the in-vehicle camera may be always in an operatingstate, and the camera may collect an image within a photographing rangebased on specific operating frequency. In addition, for an electricvehicle sensitive to power consumption, a camera may be alternativelyturned off by default, or turned on by default when there is a user inthe vehicle.

FIG. 4 is a schematic diagram of a scenario of an in-vehicle mid-airgesture according to an embodiment of this disclosure. FIG. 5 is aschematic diagram of modeling of an in-vehicle mid-air gesture accordingto an embodiment of this disclosure.

For example, because there are many users in the cabin and the usershave different motion ranges and angles, the processor may alternativelyturn off a mid-air gesture function by default, to reduce powerconsumption and save computing power. In this case, when a user expectsto interact with the vehicle-mounted machine by using a mid-air gesture,as shown in FIG. 4 , the user may input a mid-air gesture enablinggesture within the photographing range of the camera. For example, theenabling gesture may be a gesture that an index finger of the user isbent and hovers for 1s. The camera may collect a plurality of frames ofimages obtained when the index finger of the user is bent and hovers.The processor may recognize, based on the plurality of frames of imagescollected by the camera, whether the user is currently inputting theenabling gesture. For another example, the enabling gesture may be thatthe user makes a “Z” shape by using a gesture. The camera may collect aplurality of frames of images of the user making the gesture. Theprocessor recognizes, based on the plurality of frames of imagescollected by the camera, whether the user is currently inputting theenabling gesture. Because space in the vehicle is large, thephotographing range that can be collected by the camera is large, andthe enabling gesture may be alternatively extended and understood as anenabling gesture or an enabling action. For example, the enablinggesture or the enabling action may be making a semicircular shape withan arm. The camera may collect a plurality of frames of images of thegesticulation action of the user. The processor recognizes, based on theplurality of frames of images collected by the camera, whether the useris currently inputting the enabling action.

That the index finger of the user is bent and hovers for 1s is used anexample below. The camera may collect, in real time, an object imagewithin the photographing range of the camera. The processor may obtain,through wired communication or wireless communication, the imagecollected by the camera. For example, the processor obtains N frames ofimages collected by the camera within the last 1s, where N is a positiveinteger. In addition, the processor may perform modeling on a collectedobject. As shown in FIG. 5 , the processor obtains N frames of imagescollected by the camera within the last 1s. For example, an x-axis and ay-axis are established by using an upper right corner of each frame ofimage as an origin O. A pixel may be used as a unit for the x-axis andthe y-axis; or a distance unit, for example, mm or cm, may be set. Theprocessor may recognize a gesture pattern of the user in the N frames ofimages, and a reference point A in the pattern. The reference point Amay be any sampling point in the gesture pattern, and may bespecifically set and adjusted in an algorithm. For example, thereference point A may be a center point of the gesture pattern, or thereference point A may be an endpoint of the index finger of the user.

If each of the N frames of images includes a gesture pattern or aquantity of images including a gesture pattern in the N frames of imagesis greater than a preset value 1, it indicates that a palm of the userkeeps appearing within the photographing range of the camera within the1s. In addition, the processor may calculate a moving distance of thepalm of the user based on coordinates of reference points A1, A2, . . ., and An in the N frames of images. If the moving distance of the palmof the user is less than a preset value 2 (for example, the preset value2 is 50 pixel units), it indicates that the palm of the user keepshovering within the photographing range of the camera within the 1s. Inthis case, the processor may determine that the user has input thepreset enabling gesture.

It should be understood that, in a specific implementation process, inaddition to a Cartesian rectangular coordinate system, a polarcoordinate system, a spherical coordinate system, a cylindricalcoordinate system, or another coordinate system may be alternativelyused to build a model, and decompose a gesture operation to determine achange in a location of the palm or a finger of the user. This is notlimited in this application. Descriptions are provided by using anexample in which a model is built based on the spherical coordinatesystem. The camera is used as an origin. It is assumed that one of thereference points is a P point, a horizontal direction is an x-axis, avertical direction is a z-axis, where an upward direction is a positivez-axis, and a y-axis is perpendicular to a plane formed by the x-axisand the z-axis. Calculation is performed based on a distance r from theP point to the origin, an included angle θ between the positive z-axisand a connection line from the origin to the P point, and an azimuth wbetween a positive x-axis and a projection line, on an xy plane, of theconnection line from the origin to the P point. The reference point maybe a mid-air gesture or one or more sampling points in a mid-airgesture.

In some other examples, the user may alternatively trigger, in anothermanner, the vehicle to enter a mid-air gesture operation mode. Forexample, the user may wake up, by inputting a voice signal for enablinga mid-air gesture, the camera to collect a mid-air gesture input by theuser. For another example, a corresponding button (for example, avirtual button or a physical button) or an operation (for example, apress operation or a double-tap operation) may be set in thevehicle-mounted machine to enable the mid-air gesture operation mode.After detecting that the user taps the button or performs the operation,the camera may start to collect a mid-air gesture input by the user. Inaddition, a location at which the index finger of the user is bent andthat is collected by the camera for the first time may be used as astart location A1 at which the user is to subsequently input a mid-airgesture. Alternatively, the camera may detect whether the user hasperformed a preset hover gesture, and if detecting that the user hasperformed the preset hover gesture, the camera may use a hover locationof the user as a start location A1 at which the user is to subsequentlyinput a mid-air gesture.

For example, a plurality of cameras may be disposed in the vehicle, andthere may be a plurality of users in the vehicle. Therefore, to obtain abetter photographing angle, a user who initiates a mid-air gesture isprompted, based on a location of the user and a location of a camera, toinput a mid-air gesture through gesticulation to a camera with a betterangle. For example, after determining that mid-air gesture detectionprecision, of the camera, corresponding to a location at which the firstmid-air gesture is initiated is less than a first preset threshold, theprocessor may prompt, by using a prompt message, the user to initiate amid-air gesture to another camera, to increase a response speed anddetection precision for subsequent mid-air gesture-based interaction.The another camera may be a camera other than the camera, or may be acamera with a best detection angle for the user. For example, it isdetected that a passenger in a rear seat has made a mid-air gesturetoward the camera of the DMS, where the camera of the DMS is disposedabove and near the A-pillar. In this case, a detection angle of thecamera of the DMS is poor, and prompt information may be displayed on adisplay to prompt the user to make a mid-air gesture toward the cameraof the CMS, where the camera of the CMS is disposed near a rearviewmirror.

The mid-air gesture detection precision may include high-precisiondetection and low-precision detection, and the mid-air gesture detectionprecision may further include medium-precision detection. Alternatively,the mid-air gesture detection precision may include high-precisiondetection and non-high-precision detection, or low-precision detectionand non-low-precision detection. In addition, a preset threshold may beused for distinguishing.

In addition, there are a plurality of optional implementations ofdetermining whether a mid-air gesture is easy to take effect or whetherhigh-precision detection or low-precision detection is possible. Forexample, for a same camera, whether a mid-air gesture is easy to takeeffect or whether high-precision detection or low-precision detection ispossible is determined based on a distance between a location of themid-air gesture and a reference plane of the camera. Alternatively, fora same camera, whether a mid-air gesture is easy to take effect orwhether high-precision detection or low-precision detection is possibleis determined based on a projection ratio of the mid-air gesture to areference plane of the camera. For example, when the distance betweenthe location of the mid-air gesture and the reference plane of thecamera or the projection ratio of the mid-air gesture to the referenceplane of the camera is greater than a second preset threshold, it isconsidered that the mid-air gesture is easy to take effect orhigh-precision detection, or non-low-precision detection is possible;otherwise, it is considered that the mid-air gesture is not easy to takeeffect, or high-precision detection or low-precision detection isimpossible, and the mid-air gesture can be ignored. The reference planeof the camera may be further extended to a plane on which a display islocated, or the like. When a mid-air gesture is not easy to take effector non-high-precision detection or low-precision detection is possible,a prompt message may be displayed on a display in the vehicle, where theprompt message is used to prompt a user who initiates the mid-airgesture to initiate a mid-air gesture to another camera.

S302: Start, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture.

The response attribute includes one or more of a response type, aresponse level, a response priority, or a response location. The userattribute includes one or more of a user identity, a user location, or auser permission.

The response type may be a driving operation, an entertainmentoperation, an air conditioner parameter adjustment operation, or thelike. The response location may be a region corresponding to a singleuser, or may be a region corresponding to the entire cabin.

It should be understood that the location of the first user isdetermined by using N pictures that correspond to the first mid-airgesture and that are collected by the camera, where N is a positiveinteger, and the user identity of the first user may be determined basedon the location. Alternatively, matching is performed between faceinformation of an in-vehicle user and N pictures that correspond to thefirst mid-air gesture and that are collected by the camera, to determinethe user identity of the first user. The face information of thein-vehicle user may be feature point information of a user face image,or may be a user face image.

The user identity may be a driver or a passenger. Specifically, afive-seat vehicle is used as an example. The passenger may be apassenger in a co-driver seat, or may be a passenger in a rear seat. Thepassenger in a rear seat may also be classified into a passenger in arear-left seat, a passenger in a rear-right seat, and a passenger in arear-middle seat, where left and right directions are determined byusing an advancing direction of the vehicle as a reference direction.Alternatively, each seat in the vehicle is numbered, and acorrespondence between the number and the user location is preset, todetermine the user location of the first user based on the picture. Withthe foregoing solution, implementation is easy, and an amount ofcalculation is reduced.

For example, a mapping relationship between a mid-air gesture and aresponse operation may be set, and the processor may determine, based onthe obtained first mid-air gesture, a response operation correspondingto the first mid-air gesture. For example, Table 1 shows an example of amapping relationship between mid-air gestures and response operations.The processor may determine, by using an image processing technology(for example, an image recognition technology), the first mid-airgesture made by the user, and determine the first response operationbased on Table 1 and the detected first mid-air gesture.

TABLE 1 Mid-air gesture Response operation Make “W” Open a vehiclewindow Make a circle Turn on an air conditioner Make an oblique linefrom the Perform an emergency brake upper left to the lower right . . .. . .

For example, a mapping relationship between a response operation and auser permission may be set. Specifically, for example, Table 2 shows anexample of a mapping relationship between response operations andminimum user permission requirements, where a level 1 and a level 3 arerespectively minimum user permissions required for performing theresponse operations, and a larger number indicates a lower permissionlevel. That is, a user permission corresponding to the level 1 is higherthan a user permission corresponding to the level 3. User permissionsmay be set based on user identities. For example, a user permission of adriver is a level 1, a user permission of a passenger in a co-driverseat is a level 2, and a user permission of a passenger in a rear seatis a level 3. Alternatively, user permissions may be set based on agesof users. For example, a user permission of a passenger older than 18years and a driver is a level 1, a user permission of a passenger olderthan 12 years and younger than or equal to 18 years is a level 2, and auser permission of a passenger younger than or equal to 12 years is alevel 3. When the processor determines, by using the image processingtechnology (for example, the image recognition technology), that thefirst mid-air gesture made by the user is “W” and the initiation user isthe driver, the processor may determine that a corresponding responseoperation is to open a vehicle window. In addition, the user permissionof the driver meets a minimum user permission for opening the vehiclewindow. In this case, the first response operation is started inresponse to the first mid-air gesture.

TABLE 2 Response operation Minimum user permission Open a vehicle windowLevel 3 Turn on an air conditioner Level 3 Perform an emergency brakeLevel 1 . . . . . .

For example, a response attribute may be set for a response operationcorresponding to a mid-air gesture. The response attribute includes oneor more of the following attributes: a response type, a responselocation, a response permission, a response level (for example, aminimum user permission required for a response), a response priority,or the like. The response attribute may also be referred to as anoperation attribute. This is not limited in this application. Forexample, the response type may be a driving operation, an entertainmentoperation, an air conditioner adjustment operation, or an on/offoperation. The response location may be a region near the driver, apassenger region, or the like. For example, the minimum user permissionrequired for a response is as follows: The minimum user permissionrequired for performing an emergency brake is the level 1, and theminimum user permission required for turning on the air conditioner isthe level 3. For example, the response priority is as follows: Thedriver is preferentially responded to, a user with a higher userpermission is preferentially responded to, or an older user ispreferentially responded to. A user attribute may also be set for a userwho initiates a mid-air gesture. The user attribute includes one or moreof the following attributes: a user location, a user identity, or a userpermission.

After obtaining the first mid-air gesture detected by the camera, theprocessor may determine the first response operation corresponding tothe first mid-air gesture, and the first user who initiates the firstmid-air gesture, obtain the response attribute of the first responseoperation and the user attribute of the first user, determine, based onthe attribute of the first response operation and the user attribute ofthe first user, whether the first response operation matches the firstuser, and when the first response operation corresponding to the firstmid-air gesture matches the first user who initiates the first mid-airgesture, start the first response operation; otherwise, skip startingthe first response operation.

For example, to adapt to personalized requirements of differentpassengers or the driver in the vehicle, a plurality of displays may bedisposed in the vehicle, or display may be performed in differentregions of one display. For example, for a car, there is a display infront of a driver to display driving information such as a digitaldashboard, there is a display in front of a passenger in a co-driverseat to display audio and video entertainment information, and there isa display at each of the back of a seat of the driver and the back of aseat of the passenger in the co-driver seat for use by a passenger in arear seat. When a user performs adjustment by using a mid-air gesture,display may be performed on a display corresponding to the user.Specifically, the processor performs display on a corresponding displaybased on an identity of a user who initiates a mid-air gesture detectedby the camera, to avoid affecting use experience of another user.

For example, because there are many functions and scenarios that can becontrolled by using a mid-air gesture in the vehicle, one function maybe controlled by different in-vehicle users. Therefore, to ensuredriving safety, different user control priorities may be set for afunction that can be controlled by using a mid-air gesture in thevehicle. For example, for mid-air gesture-based control on up/down offour vehicle windows in the vehicle, the driver may be set to have ahighest user control priority. An up/down setting of a vehicle window ofthe driver seat can be controlled only by the driver. User controlpriorities for up/down of a vehicle window near another passenger are asfollows: the driver, a passenger identity corresponding to the vehiclewindow, and a passenger identity corresponding to another passenger.Alternatively, for the mid-air gesture-based control on the up/down ofthe four vehicle windows in the vehicle, user control priorities are setbased on ages. A user older than 10 years may control up/down of avehicle window in the vehicle, and a user younger than or equal to 10years is not granted a control permission. For another example, formid-air gesture-based control on the digital dashboard, the driver maybe set to have a highest user control priority, and setting may beperformed only by the driver. Another user cannot control a function ofthe digital dashboard regardless of how the user initiates a mid-airgesture, thereby ensuring driving safety. Similarly, in a same timeperiod, if a plurality of users in the vehicle perform mid-air gestures,when computing power is limited, a mid-air gesture initiated by a usercorresponding to a higher control priority may be preferentiallyprocessed. For example, in the time period, only the mid-air gestureinitiated by the user is processed; or the mid-air gesture initiated bythe user is processed first, and then a mid-air gesture initiated byanother user is processed. The control priority may be set beforedelivery, or may be set by the user during use.

Assuming that the first mid-air gesture is not an enabling gesture,before S301, the method 300 may further include S303.

S303: Obtain an enabling gesture detected by the camera, and display apreset indication icon on an in-vehicle display in response to theenabling gesture.

In step S303, after the processor detects that the user has input thepreset enabling gesture, it indicates that the user expects to control,by using a mid-air gesture, an application displayed on a currentin-vehicle display. To guide and prompt the user to efficiently andcorrectly input a mid-air gesture, a preset indication icon may bedisplayed at a preset location on an in-vehicle display, and a location,a length, or a shape of the indication icon on an interface arechangeable.

FIG. 6 is a schematic diagram of a scenario of an in-vehicle mid-airgesture according to an embodiment of this disclosure. For example, amusic app is running on a display of the vehicle-mounted machine. Asshown in FIG. 6 , when the user inputs the enabling gesture, the displayis displaying a first display interface of the music app. After it isdetected that the user has input the enabling gesture, a firstindication icon may be displayed on the display. After the firstindication icon appears on the first display interface of the music app,the user may learn that the vehicle-mounted machine has entered themid-air gesture operation mode, so that the user is prompted to start toinput a corresponding mid-air gesture.

The first indication icon 602 may be a swipe indication icon, a progressbar, or the like. The first indication icon 602 may be a linear orarc-shaped icon. A display effect, for example, a specific shape, size,location, or color, of the first indication icon 602 is not limited inthis embodiment of this disclosure. The first indication icon 602 may bestatic or dynamic.

Alternatively, assuming that the first mid-air gesture is not anenabling gesture, before S301, the method 300 may further include S304.

S304: Obtain an enabling gesture detected by the camera, and obtain, inresponse to the enabling gesture, the first mid-air gesture that isinput by the user and that is detected by the camera.

In step S304, after detecting that the user has input the presetenabling gesture, the processor determines that the user expects tocontrol, by using a mid-air gesture, an application running in thevehicle or a controllable function, and enters the mid-air gestureoperation mode. In this case, the processor may continue to collect, byusing the camera, each frame of in-vehicle image in real time, andrecognize, based on the collected image, a specific mid-air gestureinput by the user.

For example, after detecting that the user has input the preset enablinggesture, the processor may use, as a start location A1, a location, of apalm or a finger of the user, that is collected when the enablinggesture ends, and then determine, in real time based on M frames ofimages that are recently collected, the first mid-air gesture input bythe user.

In addition, a difference from a case in which a user uses a mobilephone lies in that the in-vehicle camera can obtain a largerphotographing range, and a location of an arm of the user can beconsidered. For example, a mid-air gesture may be alternatively amid-air action or making a circle with an arm.

For example, a mid-air gesture is recognized by using a detected keypoint, to recognize that the user has made a single-finger-up gesture, atwo-finger gesture, or the like. Then whether the mid-air gesture is amid-air gesture related to an application on a current interface isdetermined by querying a database or the like. If the mid-air gesture isa gesture corresponding to a function of the application on the currentinterface, the function is used as a target function. If the mid-airgesture is not any function of the application on the current interface,the display does not respond, or prompts the user that the mid-airgesture is incorrect.

With the setting of the enabling gesture, indiscriminate response of theprocessor caused by an unintentional action of an in-vehicle user can beeffectively avoided, thereby ensuring functional stability in thevehicle and driving safety.

The method 300 may further include S305.

S305: In response to the first mid-air gesture, the indication icondisplayed on the display moves based on the moving trajectory of thefirst mid-air gesture.

For example, a space-based adjustment manner is a manner of determiningan adjustment amount based on a first distance at which a mid-airgesture moves within a photographing range. In this manner, acontinuously adjusted adjustment amount is positively correlated withthe first distance at which the mid-air gesture moves within thephotographing range. A time-based adjustment manner is a manner ofdetermining an adjustment amount based on holding duration of a mid-airgesture within a photographing range. In this manner, a continuouslyadjusted adjustment amount is positively correlated with the holdingduration of the mid-air gesture within the photographing range. Thespace-based adjustment manner or the time-based adjustment manner may beset before delivery, or may be open for autonomous setting by the user.

S306: In response to the first mid-air gesture, display, on the display,a second display interface corresponding to the first mid-air gesture.

For example, when the first response operation corresponding to thefirst mid-air gesture matches the first user who initiates the firstmid-air gesture and the first display is not a display in front of thefirst user, the first display interface is displayed on a second displayin the vehicle in response to the first mid-air gesture, and anindication icon on the first display interface is moved based on amoving trajectory of the first mid-air gesture in a preset direction.

FIG. 7 is a schematic diagram of a structure of an electronic apparatusaccording to an embodiment of this disclosure. It should be understoodthat the electronic apparatus 700 may be a terminal, for example, avehicle or a vehicle-mounted machine, or may be a built-in chip of aterminal, and may implement the mid-air gesture-based interaction methodshown in FIG. 3 and the foregoing optional embodiments. As shown in FIG.7 , the electronic apparatus 700 includes a processor 701 and aninterface circuit 702 coupled to the processor. It should be understoodthat, although only one processor and one interface circuit are shown inFIG. 7 , the image processing apparatus 700 may include another quantityof processors and interface circuits.

The interface circuit 702 is connected to another component of theterminal, for example, a memory or another processor. The processor 701is configured to perform signal interaction with another component byusing the interface circuit 702. The interface circuit 702 may be aninput/output interface of the processor 701.

The processor 701 may be a processor in a vehicle-mounted device, forexample, a vehicle-mounted machine, or may be a processing apparatussold separately.

For example, the processor 701 reads, by using the interface circuit702, computer programs or instructions in the memory coupled to theprocessor 701, and decodes and executes the computer programs orinstructions. It should be understood that the computer programs or theinstructions may include the foregoing terminal function programs, ormay include the foregoing mid-air gesture-based interaction functionprogram used on a terminal. When a corresponding functional program isdecoded and executed by the processor 701, the terminal or theelectronic apparatus in the terminal can be enabled to implement thesolution in the image processing method provided in embodiments of thisdisclosure.

Optionally, the terminal function programs are stored in an externalmemory of the electronic apparatus 700. When the terminal functionprograms are decoded and executed by the processor 701, some or allcontent of the terminal function programs is temporarily stored in thememory.

Optionally, the terminal function programs are stored in an internalmemory of the electronic apparatus 700. When the terminal functionprograms are stored in the internal memory of the electronic apparatus700, the electronic apparatus 700 may be disposed in the terminal inthis embodiment of the present invention.

Optionally, some content of the terminal function programs is stored inan external memory the electronic apparatus 700, and other content ofthe terminal function programs is stored in an internal memory of theelectronic apparatus 700.

FIG. 8 is a schematic diagram of a structure of another electronicapparatus according to an embodiment of this disclosure. It should beunderstood that the electronic apparatus 800 may be a terminal, forexample, a vehicle or a vehicle-mounted machine, or may be a built-inchip of a terminal, and may implement the mid-air gesture-basedinteraction method shown in FIG. 3 and the foregoing optionalembodiments. As shown in FIG. 8 , the electronic apparatus 800 includesa processing unit 801 and a transceiver unit 802 coupled to theprocessor. It should be understood that, although only one processingunit and one transceiver unit are shown in FIG. 8 , the image processingapparatus 800 may include another quantity of processing units andtransceiver units.

The processing unit and the transceiver unit may be implemented byhardware or software. The processing unit 801 may be the processor 701in FIG. 7 , and the transceiver unit 802 may be the interface circuit702 in FIG. 7 .

The electronic apparatus 800 may be configured to perform the methodshown in FIG. 3 and the methods in the optional embodiments.

FIG. 9 is a schematic diagram of a structure of another electronicapparatus according to an embodiment of this disclosure. It should beunderstood that the electronic apparatus 800 may be a terminal, forexample, a vehicle or a vehicle-mounted machine, or may be a built-inchip of a terminal, and may implement the mid-air gesture-basedinteraction method shown in FIG. 3 and the foregoing optionalembodiments. As shown in FIG. 9 , the electronic apparatus 900 includesa processor 901 and a memory 902 coupled to the processor 901. It shouldbe understood that, although only one processor and one memory are shownin FIG. 9 , the electronic apparatus 901 may include another quantity ofprocessors and memories.

The memory 902 is configured to store computer program or computerinstructions. The computer programs or the instructions may beclassified into two types based on functions. When a type of computerprogram or instruction is executed by the processor 901, the electronicapparatus 90 is enabled to implement the steps of the terminal in thewireless communication method in embodiments of the present invention.This type of computer program or instruction may be denoted as aterminal function program. For example, the terminal function programmay include program code for implementing the in-vehicle mid-airgesture-based interaction method shown in FIG. 3 .

In addition, the electronic apparatus 90 may further include aconnection line 900, a transmit circuit 903, a receive circuit 904, anantenna 905, an input/output (input/output, I/O) interface 906, and thelike. The transmit circuit and the receive circuit may be coupled to theantenna, and are wirelessly connected to another communications device.The transmit circuit and the receive circuit may be alternativelyintegrated into one transceiver, and the antenna may be a radiofrequency antenna supporting a plurality of frequencies. The I/Ointerface provides a possibility of interacting with anothercommunications device or a user. For example, the I/O interface may bean Ethernet interface or a USB interface. For a terminal, the I/Ointerface may be a screen, a keyboard, a microphone, a speaker, or a USBinterface. The components in the electronic apparatus 90 may be coupledtogether by using various connection lines (for example, a bus system).In addition to a data bus, the bus system may further include a powerbus, a control bus, a status signal bus, and the like. However, forclear description, various types of buses in this specification arecollectively referred to as the bus system.

It can be understood that the processor 901 and the memory 902 may beimplemented by using a processing unit and a storage unit instead. Theprocessing unit and the storage unit may be implemented by using codewith corresponding functions. The storage unit is configured to storeprogram instructions. The processing unit is configured to execute theprogram instructions in the storage unit, to implement the method shownin FIG. 3 and the foregoing optional embodiments.

It should be understood that the electronic apparatuses shown in any oneof FIG. 1 , FIG. 2A, and FIG. 7 to FIG. 9 may be combined with eachother. For related design details of the optional embodiments, refer tothe optional embodiments, or refer to related design details of themethod shown in FIG. 3 and the optional embodiments. Details are notdescribed herein again.

Based on the foregoing descriptions of the implementations, a personskilled in the art may clearly understand that, for the purpose ofconvenient and brief descriptions, division into the foregoing functionmodules is merely used as an example for descriptions. During actualapplication, the foregoing functions may be allocated to differentfunction modules for implementation as required. In other words, aninner structure of an apparatus is divided into different functionmodules to implement all or some of the functions described above. For aspecific working process of the foregoing system, apparatus, and unit,refer to a corresponding process in the foregoing method embodiments.Details are not described herein again.

The terms “first”, “second”, “third”, “fourth”, and the like inembodiments and accompanying drawings are used to distinguish betweensimilar objects, but not necessarily used to describe a specific orderor sequence. In addition, the terms “comprise”, “include”, and anyvariants thereof are intended to indicate a non-exclusive inclusion. Forexample, a series of steps or units are included. A method, a system, aproduct, or a device is not necessarily limited to listed steps orunits, but may include other steps or units that are not listed and thatare inherent to the process, the method, the product, or the device.

It should be understood that, in this disclosure, “at least one” meansone or more, and “a plurality of” means two or more. A term “and/or” isused for describing an association relationship between associatedobjects, and represents that three relationships may exist. For example,“A and/or B” may represent the following three cases: Only A exists,only B exists, and both A and B exist, where A and B may be singular orplural. A character “/” usually indicates an “or” relationship betweenassociated objects. In addition, “at least one of the following items(pieces)” or a similar expression thereof indicates any combination ofthese items, including a single item (piece) or any combination of aplurality of items (pieces). For example, at least one of a, b, or c mayindicate a, b, c, “a and b”, “a and c”, “b and c”, or “a, b, and c”,where a, b, and c may be singular or plural.

It should be understood that sequence numbers of the foregoing processesdo not mean execution sequences in this application. The executionsequences of the processes should be determined according to functionsand internal logic of the processes, and should not be construed as anylimitation on implementation processes of embodiments of thisdisclosure. The term “coupling” mentioned in this disclosure is used toexpress interworking or interaction between different components, andmay include direct connection or indirect connection by using anothercomponent.

Function units in embodiments of this disclosure may be integrated intoone processing unit, or each of the units may exist alone physically, ortwo or more units are integrated into one unit. The integrated unit maybe implemented in a form of hardware, or may be implemented in a form ofa software function unit.

All or some of embodiments of this disclosure may be implemented byusing software, hardware, firmware, or any combination thereof. Whensoftware is used to implement embodiments, all or some of embodimentsmay be implemented in a form of a computer program product. The computerprogram product includes one or more computer instructions. When thecomputer program instructions are loaded and executed on a computer, allor some of the procedures or functions according to embodiments of thisdisclosure are generated. The computer may be a general-purposecomputer, a special-purpose computer, a computer network, or anotherprogrammable apparatus. The computer instructions may be stored in acomputer-readable storage medium or may be transmitted from acomputer-readable storage medium to another computer-readable storagemedium. For example, the computer instructions may be transmitted from awebsite, computer, server, or data center to another website, computer,server, or data center in a wired (for example, a coaxial cable or anoptical fiber) or wireless (for example, infrared, radio, or microwave)manner. The computer-readable storage medium may be any usable mediumaccessible by the computer, or a data storage device, for example, aserver or a data center, integrating one or more usable media. Theusable medium may be a magnetic medium, for example, a floppy disk, ahard disk, or a magnetic tape, may be an optical medium, for example, aDVD, or may be a semiconductor medium, for example, a solid-state drive(solid-state drive, SSD).

In embodiments of this disclosure, the memory is a device or circuitwith a data or information storage capability, and may provideinstructions and data for the processor. The memory includes a read-onlymemory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a nonvolatile random access memory (NVRAM), aprogrammable read-only memory, an electrically erasable programmablememory, a register, or the like.

The foregoing descriptions are merely specific implementations ofembodiments of this disclosure, but are not intended to limit theprotection scope of embodiments of this disclosure. Any variation orreplacement within the technical scope disclosed in embodiments of thisdisclosure shall fall within the protection scope of embodiments of thisdisclosure. Therefore, the protection scope of embodiments of thisdisclosure shall be subject to the protection scope of the claims.

What is claimed is:
 1. A method for operating a vehicle, comprising:recording, by a camera, a mid-air gesture; comparing the recording to aplurality of stored gestures; determining a stored gesture correspondsto the recording; and initiating a first response that corresponds tothe stored mid-air gesture to modify an operation of the vehicle.
 2. Themethod according to claim 1, wherein before initiating the firstresponse, the method further comprises: obtaining a response attributeof the first response operation corresponding to the first mid-airgesture; obtaining a user attribute of the first user who initiates thefirst mid-air gesture; and determining, based on the response attributeof the first response operation and the user attribute of the firstuser, whether the first response operation matches the first user,wherein the response attribute comprises one or more of a response type,a response level, a response priority, or a response location, and theuser attribute comprises one or more of a user identity, a userlocation, or a user permission.
 3. The method according to claim 1,wherein the obtaining a user attribute of the first user specificallycomprises: determining the user location of the first user based on apicture that corresponds to the first mid-air gesture and that isdetected by the camera; and determining the user identity of the firstuser based on the user location of the first user.
 4. The methodaccording to claim 1, wherein when the response attribute of the firstresponse operation is a driving operation and the user identity of thefirst user is a driver, the first response operation matches the firstuser.
 5. The method according to claim 1, wherein the obtaining a userattribute of the first user specifically comprises: determining the userlocation of the first user based on a picture that corresponds to thefirst mid-air gesture and that is detected by the camera; and thestarting, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture specifically comprises: starting, when the first responseoperation corresponding to the first mid-air gesture matches the firstuser who initiates the first mid-air gesture, the first responseoperation based on the user location of the first user in response tothe first mid-air gesture.
 6. The method according to claim 5, whereinthe starting, when the first response operation corresponding to thefirst mid-air gesture matches the first user who initiates the firstmid-air gesture, the first response operation based on the user locationof the first user in response to the first mid-air gesture specificallycomprises: starting, when the first response operation corresponding tothe first mid-air gesture matches the first user who initiates the firstmid-air gesture, the first response operation in a region correspondingto the user location of the first user in response to the first mid-airgesture.
 7. The method according to claim 5, wherein before thestarting, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture, the method further comprises: displaying a first displayinterface on a first display in a vehicle; and the starting, when thefirst response operation corresponding to the first mid-air gesturematches the first user who initiates the first mid-air gesture, thefirst response operation based on the user location of the first user inresponse to the first mid-air gesture specifically comprises: when thefirst response operation corresponding to the first mid-air gesturematches the first user who initiates the first mid-air gesture and thefirst display is not a display in front of the first user, displayingthe first display interface on a second display in the vehicle inresponse to the first mid-air gesture, and moving an indication icon onthe first display interface based on a moving trajectory of the firstmid-air gesture in a preset direction.
 8. The method according to claim1, wherein the starting, when a first response operation correspondingto the first mid-air gesture matches a first user who initiates thefirst mid-air gesture, the first response operation in response to thefirst mid-air gesture specifically comprises: when a user permissionthreshold required for the first response operation corresponding to thefirst mid-air gesture is less than or equal to the user permission ofthe first user who initiates the first mid-air gesture, starting thefirst response operation in response to the first mid-air gesture. 9.The method according to claim 1, wherein the method further comprises:obtaining a second mid-air gesture detected by the camera, wherein thesecond mid-air gesture is initiated by a second user, and durationbetween an initiation time of the second mid-air gesture and aninitiation time of the first mid-air gesture is less than a first presetthreshold; and the starting, when a first response operationcorresponding to the first mid-air gesture matches a first user whoinitiates the first mid-air gesture, the first response operation inresponse to the first mid-air gesture specifically comprises: when theuser permission of the first user is higher than a user permission ofthe second user, and when the first response operation corresponding tothe first mid-air gesture matches the first user who initiates the firstmid-air gesture, preferentially starting the first response operation inresponse to the first mid-air gesture.
 10. The method according to claim8, wherein a control permission of the driver is higher than a controlpermission of another passenger.
 11. The method according to claim 1,wherein before the starting, when a first response operationcorresponding to the first mid-air gesture matches a first user whoinitiates the first mid-air gesture, the first response operation inresponse to the first mid-air gesture, the method further comprises:when a distance between a location of the first mid-air gesture and thecamera is greater than a second preset threshold, displaying a promptmessage on a display in the vehicle, wherein the prompt message is usedto prompt the first user to initiate a mid-air gesture to anothercamera.
 12. The method according to claim 1, wherein before thestarting, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture, the method further comprises: when a projection ratio of thefirst mid-air gesture relative to a reference plane of a first camera isless than a third preset threshold, displaying a prompt message on adisplay in the vehicle, wherein the prompt message is used to prompt thefirst user to initiate a mid-air gesture to another camera.
 13. Anelectronic apparatus, comprising: a interface circuit; and a processor,wherein: the processor is configured to communicate with a camera toreceive image data that corresponds to a first mid-air gesture detectedby the camera; and the processor is configured to process the image dataand to initiate a first response operation when a first responseoperation corresponding to the first mid-air gesture matches a firstuser who initiates the first mid-air gesture.
 14. The electronicapparatus according to claim 13, wherein the processor is furtherconfigured to obtain a response attribute of the first responseoperation corresponding to the first mid-air gesture; the processor isfurther configured to obtain a user attribute of the first user whoinitiates the first mid-air gesture; and the processor is furtherconfigured to determine, based on the response attribute of the firstresponse operation and the user attribute of the first user, whether thefirst response operation matches the first user, wherein the responseattribute comprises one or more of a response type, a response level, aresponse priority, or a response location, and the user attributecomprises one or more of a user identity, a user location, or a userpermission.
 15. The electronic apparatus according to claim 13, whereinthe processor is further configured to determine the user location ofthe first user based on a picture that corresponds to the first mid-airgesture and that is detected by the camera; and the processor is furtherconfigured to determine the user identity of the first user based on theuser location of the first user.
 16. The electronic apparatus accordingto claim 13, wherein when the response attribute of the first responseoperation is a driving operation and the user identity of the first useris a driver, the first response operation matches the first user. 17.The electronic apparatus according to claim 13, wherein the processor isfurther configured to determine the user location of the first userbased on a picture that corresponds to the first mid-air gesture andthat is detected by the camera; and that the processor is configured tostart, when a first response operation corresponding to the firstmid-air gesture matches a first user who initiates the first mid-airgesture, the first response operation in response to the first mid-airgesture specifically means that: the processor is configured to start,when the first response operation corresponding to the first mid-airgesture matches the first user who initiates the first mid-air gesture,the first response operation based on the user location of the firstuser in response to the first mid-air gesture.
 18. The electronicapparatus according to claim 17, wherein that the processor isconfigured to start, when the first response operation corresponding tothe first mid-air gesture matches the first user who initiates the firstmid-air gesture, the first response operation based on the user locationof the first user in response to the first mid-air gesture specificallycomprises: the processor is configured to start, when the first responseoperation corresponding to the first mid-air gesture matches the firstuser who initiates the first mid-air gesture, the first responseoperation in a region corresponding to the user location of the firstuser in response to the first mid-air gesture.
 19. The electronicapparatus according to claim 17, wherein the processor is furtherconfigured to display a first display interface on a first display in avehicle; and that the processor is configured to start, when a firstresponse operation corresponding to the first mid-air gesture matches afirst user who initiates the first mid-air gesture, the first responseoperation in response to the first mid-air gesture specifically meansthat: the processor is configured to: when the first response operationcorresponding to the first mid-air gesture matches the first user whoinitiates the first mid-air gesture and the first display is not adisplay in front of the first user, display the first display interfaceon a second display in the vehicle in response to the first mid-airgesture, and move an indication icon on the first display interfacebased on a moving trajectory of the first mid-air gesture in a presetdirection.
 20. A computer program product, comprising: computerinstructions, which when executed by a processor or a device, causes thedevice to perform the steps of: recording, by a camera, a mid-airgesture; comparing the recording to a plurality of stored gestures;determining a stored gesture corresponds to the recording; andinitiating a first response that corresponds to the stored mid-airgesture to modify an operation of the vehicle.